Image acquisition device and method of controlling the same

ABSTRACT

Provided is an artificial intelligence (AI) system that mimics functions, such as recognition and determination by human brains, by utilizing a machine learning algorithm, such as deep learning, and applications of the AI system. An image acquisition device is disclosed including a camera configured to acquire a first image, wherein a portion of a main object is hidden from the camera by a sub-object; at least one processor configured to input the first image to a first AI neural network; detect, by the first AI neural network from data corresponding to a plurality of objects included in the first image, first data corresponding to the main object and second data corresponding to the sub-object from the first image by inputting the first image to an AI neural network, remove the sub-object from the first image, and generate, using a second AI neural network, a second image by restoring third data corresponding to at least a portion of the main object hidden by the removed sub-object by using the AI neural network, wherein the third data replaces the second data; and a display configured to display at least one of the first image and the second image.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a Continuation Application of U.S. application Ser.No. 16/232,711 filed on Dec. 26, 2018, which is based on and claimspriority under 35 U.S.C. § 119 to Korean Patent Application No.10-2017-0180036, filed on Dec. 26, 2017 and Korean Patent ApplicationNo. 10-2018-0169526, filed on Dec. 26, 2018, in the Korean IntellectualProperty Office, the disclosures of which are incorporated by referenceherein in their entirety.

BACKGROUND 1. Field

One or more embodiments relate to an image acquisition device and amethod of controlling the same.

2. Description of Related Art

Artificial intelligence (AI) systems are computer systems configured torealize human-level intelligence and train themselves and makedeterminations spontaneously to become smarter, in contrast to existingrule-based smart systems. Since recognition rates of AI systems haveimproved and the AI systems more accurately understand a user'spreferences the more they are used, existing rule-based smart systemsare being gradually replaced by deep-learning AI systems.

AI technology includes machine learning (deep learning) and elementtechnologies employing the machine learning.

The machine learning is algorithm technology that self-classifies/learnsthe characteristics of input data, and each of the element technologiesis technology using a machine learning algorithm, such as deep learning,and includes technical fields, such as linguistic understanding, visualunderstanding, deduction/prediction, knowledge representation, andoperation control.

Various fields to which AI technology is applied are as follows.Linguistic understanding is a technique of recognizing alanguage/character of a human and applying/processing thelanguage/character of a human, and includes natural language processing,machine translation, a conversation system, questions and answers, voicerecognition/synthesis, and the like. Visual understanding is a techniqueof recognizing and processing an object like in human vision, andincludes object recognition, object tracking, image search, humanrecognition, scene understanding, space understanding, imageimprovement, and the like.

Deduction/prediction is technology of logically performing deduction andprediction by determining information, and includesknowledge/probability-based deduction, optimization prediction, apreference-based plan, recommendation, and the like. Knowledgerepresentation is a technique of automatically processing humanexperience information as knowledge data, and includes knowledgeestablishment (data generation/classification), knowledge management(data utilization), and the like. Operation control is a technique ofcontrolling autonomous driving of a vehicle and motions of a robot, andincludes motion control (navigation, collision avoidance, and driving),manipulation control (behavior control), and the like.

AI technology may also be used to acquire images, such as pictures ormoving pictures.

SUMMARY

Provided is an image acquisition device that distinguishes a main objectwithin an image from a sub-object within the image.

Provided is an image acquisition device that displays an image such thata user may easily recognize a sub-object within an image.

Provided is an image acquisition device that removes a sub-object froman image and displays an image from which the sub-object has beenremoved.

Provided is an image acquisition device that acquires an image in whichat least a portion of a main object hidden by a sub-object has beenrestored.

Additional aspects will be set forth in part in the description whichfollows and, in part, will be apparent from the description, or may belearned by practice of the presented embodiments.

In accordance with an aspect of this disclosure, a method is provided ofacquiring an image by using an artificial intelligence (AI) neuralnetwork, the method including: acquiring a first image by using acamera, wherein a portion of a main object is hidden from the camera bya sub-object; inputting the first image to a first AI neural network;detecting, by the first AI neural network, from data corresponding to aplurality of object included the first image, first data correspondingto the main object and second data corresponding to the sub-object;removing the second data corresponding to the sub-object from the firstimage; and generating, using a second AI neural network, a second imageby restoring third data corresponding to at least a portion of the mainobject hidden by the sub-object, wherein the third data replaces thesecond data.

In some embodiments of the method, the detecting of the first datacorresponding to the main object and the second data corresponding tothe sub-object comprises displaying an indicator indicating the seconddata corresponding to the sub-object together with the first image.

In some embodiments of the method, the restoring of the third datacorresponding to the at least the portion of the main object comprises:acquiring information of first relative locations between the camera andthe main object; and restoring the third data corresponding to the atleast the portion of the main object, based on the information of thefirst relative locations.

In some embodiments, the method includes determining a first sharpnessof the third data corresponding to the at least the portion of the mainobject; and performing image processing such that a second sharpness ofthe third data corresponding to the at least the portion of the mainobject corresponds to the first sharpness.

In some embodiments of the method, the detecting of the first datacorresponding to the main object and the second data corresponding tothe sub-object comprises: receiving a user selection of the sub-objectfrom the first image; and detecting the second data corresponding to thesub-object from the first image, based on the user selection.

In some embodiments of the method, the displaying of the indicatorincludes: tracking a motion of the sub-object; and displaying theindicator, based on the motion of the sub-object.

In some embodiments of the method, the tracking of the motion of thesub-object includes: acquiring information of second relative locationsbetween the camera and the sub-object; inputting the information of thesecond relative locations to the first AI neural network; and tracking,by the first AI neural network, the motion of the sub-object bydetecting a change in the information of the second relative locations.

In some embodiments of the method, the detecting of the first datacorresponding to the main object and the second data corresponding tothe sub-object includes: forming, in response to a user input of drivingthe camera, a communication link with a server, wherein the serverincludes the first AI neural network and the second AI neural network;transmitting, over the communication link, the first image to theserver; and receiving, from the server over the communication link,information about a result of detecting the first data corresponding tothe main object and the second data corresponding to the sub-object.

In some embodiments, the method includes receiving, from a server and inresponse to a user input of driving the camera, data for updating atleast one of the first AI neural network and the second AI neuralnetwork.

In some embodiments of the method, the first AI neural network is amodel trained to detect the first data corresponding to the main objectby using at least one pre-stored image as learning data.

In accordance with an aspect of this disclosure, a image acquisitiondevice is provided.

In some embodiments, the image acquisition device includes a cameraconfigured to acquire a first image, wherein a portion of a main objectis hidden from the camera by a sub-object. In some embodiments, theimage acquisition device also includes at least one processor configuredto: input the first image to a first AI neural network; detect, by thefirst AI neural network from data corresponding to a plurality of objectincluded the first image, first data corresponding to the main objectand second data corresponding to the sub-object, and generate, using asecond AI neural network, a second image by restoring third datacorresponding to at least a portion of the main object hidden by thesub-object, wherein the third data replaces the second data. The imageacquisition device also includes a display configured to display atleast one of the first image and the second image.

In some embodiments of the image acquisition device, the display isfurther configured to display an indicator indicating the second datacorresponding to the sub-object, together with the first image.

In some embodiments of the image acquisition device, the at least oneprocessor is further configured to: acquire information of firstrelative locations between the camera and the main object; and restorethe third data corresponding to the at least a portion of the mainobject, based on the information of the first relative locations.

In some embodiments of the image acquisition device, the at least oneprocessor is further configured to: determine a first sharpness of thethird data corresponding to the at least portion of the main object; andperform image processing such that a second sharpness of the third datacorresponding to the at least portion of the main object corresponds tothe first sharpness.

In some embodiments, the image acquisition device also includes a userinput interface configured to receive user selection of the sub-objectfrom the first image, wherein the at least one processor is furtherconfigured to detect the second data corresponding to the sub-object,based on the user selection.

In some embodiments of the image acquisition device, the at least oneprocessor is further configured to track a motion of the sub-object, andthe display is further configured to display the indicator, based on themotion of the sub-object.

In some embodiments of the image acquisition device, the at least oneprocessor is further configured to acquire information of secondrelative locations between the camera and the sub-object; input theinformation of the second relative locations to the first AI neuralnetwork; and track the motion of the sub-object by detecting a change inthe information of the second relative locations.

In some embodiments, the image acquisition device also includes a userinput interface configured to receive a user input of driving thecamera. In some embodiments, the image acquisition device also includesa communication interface configured to: form, in response to the userinput, a communication link with a server including the first AI neuralnetwork and the second AI neural network, transmit the first image tothe server, and receive, from the server, information about a result ofdetecting the first data corresponding to the main object and the seconddata corresponding to the sub-object from the first image.

In some embodiments, the image acquisition device also includes a userinput interface configured to receive a user input of driving thecamera. In some embodiments, the image acquisition device also includesa communication interface configured to: form, in response to the userinput, a communication link with a server that updates at least one ofthe first AI neural network and the second AI neural network andreceive, over the communication link, data for updating at least one ofthe first AI neural network and the second AI neural network from theserver.

In some embodiments of the image acquisition device, the first AI neuralnetwork is a model trained to detect the first data corresponding to themain object by using at least one pre-stored image as learning data.

In accordance with yet another aspect of the disclosure, a method ofacquiring an image by using an artificial intelligence (AI) neuralnetwork includes acquiring a first image by using a camera; detecting amain object and a sub-object from the first image by inputting the firstimage to the AI neural network; removing the sub-object from the firstimage; and generating a second image by restoring at least a portion ofthe main object hidden by the removed sub-object, by using the AI neuralnetwork.

In accordance with yet another aspect of the disclosure, an imageacquisition device includes a camera configured to acquire a firstimage; at least one processor configured to detect a main object and asub-object from the first image by inputting the first image to an AIneural network, remove the sub-object from the first image, and generatea second image by restoring at least a portion of the main object hiddenby the removed sub-object by using the AI neural network; and a displayconfigured to display at least one of the first image and the secondimage.

In accordance with yet another aspect of the disclosure, a computerprogram product including a non-transitory computer-readable storagemedium has recorded thereon a method of acquiring an image by using anAI neural network, the non-transitory computer-readable storage mediumincluding instructions to acquire a first image by using a camera;detect a main object and a sub-object from the first image by inputtingthe first image to the AI neural network; remove the sub-object from thefirst image; and generate a second image by restoring at least a portionof the main object hidden by the removed sub-object, by using the AIneural network.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects, features, and advantages of certainembodiments of the present disclosure will be more apparent from thefollowing description taken in conjunction with the accompanyingdrawings, in which:

FIG. 1 illustrates an example in which an image acquisition deviceacquires an image, according to some embodiments;

FIG. 2 is a flowchart of a method, performed by the image acquisitiondevice, of acquiring an image, according to some embodiments;

FIGS. 3, 4 and 5 are views illustrating examples of displaying anindicator indicating a sub-object, according to some embodiments;

FIG. 6 is a flowchart of a method of restoring at least a portion of amain object hidden by a sub-object, according to some embodiments;

FIGS. 7, 8, 9, 10, and 11 illustrate examples of restoring at least aportion of a main object hidden by a sub-object, according to someembodiments;

FIG. 12 is a flowchart of a method of detecting a sub-object in responseto a user input, according to some embodiments;

FIGS. 13 and 14 illustrate examples of detecting a sub-object inresponse to a user input, according to some embodiments;

FIG. 15 is a flowchart of a method of restoring at least a portion of amain object hidden by a sub-object in response to a user input,according to some embodiments;

FIGS. 16, 17 and 18 illustrate examples of restoring at least a portionof a main object hidden by a sub-object, in response to a user input,according to some embodiments;

FIG. 19 is a flowchart of a method of tracking a motion of a sub-objectwithin an image, according to some embodiments;

FIGS. 20, 21, 22, and 23 are views illustrating examples of tracking amotion of a sub-object within an image, according to some embodiments;

FIG. 24 is a flowchart of a method of restoring at least a portion of amain object hidden by a sub-object, based on relative locations betweena camera and at least one of the main object and the sub-object,according to some embodiments;

FIGS. 25 and 26 illustrate examples of detecting a main object and asub-object when the camera moves, according to some embodiments;

FIG. 27 is a view illustrating a method of displaying an indicatorindicating a sub-object by using a server, according to someembodiments;

FIGS. 28 and 29 are views illustrating examples of displaying anindicator indicating a sub-object by using the server, according to someembodiments;

FIGS. 30 and 31 are block diagrams of the image acquisition deviceaccording to some embodiments;

FIG. 32 is a block diagram of the server according to some embodiments;

FIG. 33 is a block diagram of a processor included in the imageacquisition device, according to some embodiments;

FIG. 34 is a block diagram of a data learner included in the processor,according to some embodiments;

FIG. 35 is a block diagram of a data recognizer included in theprocessor, according to some embodiments; and

FIG. 36 is a block diagram illustrating an example where the imageacquisition device and the server interoperate to learn and recognizedata, according to some embodiments.

DETAILED DESCRIPTION

Embodiments of the present disclosure are described in detail hereinwith reference to the accompanying drawings so that this disclosure maybe easily performed by one of ordinary skill in the art to which thepresent disclosure pertains. The disclosure may, however, be embodied inmany different forms and should not be construed as being limited to theembodiments set forth herein. In the drawings, parts irrelevant to thedescription are omitted for simplicity of explanation, and like numbersrefer to like elements throughout.

The aforementioned embodiments may be described in terms of functionalblock components and various processing steps. Some or all of suchfunctional blocks may be realized by any number of hardware and/orsoftware components configured to perform the specified functions. Forexample, functional blocks according to the disclosure may be realizedby one or more microprocessors or by circuit components for apredetermined function. In addition, for example, functional blocksaccording to the disclosure may be implemented with any programming orscripting language. The functional blocks may be implemented inalgorithms that are executed on one or more processors. Furthermore, thedisclosure described herein could employ any number of conventionaltechniques for electronics configuration, signal processing and/orcontrol, data processing and the like. The words “mechanism,” “element,”“means,” and “configuration” are used broadly and are not limited tomechanical or physical embodiments,

Throughout the specification, when an element is referred to as being“connected” or “coupled” to another element, it can be directlyconnected or coupled to the other element, or can be electricallyconnected or coupled to the other element with intervening elementsinterposed therebetween. In addition, the terms “comprises” and/or“comprising” or “includes” and/or “including” when used in thisspecification, specify the presence of stated elements, but do notpreclude the presence or addition of one or more other elements.

Furthermore, the connecting lines or connectors between components shownin the various figures presented are intended to represent exemplaryfunctional relationships and/or physical or logical couplings betweenthe components. Connections between components may be represented bymany alternative or additional functional relationships, physicalconnections or logical connections in a practical device.

While such terms as “first,” “second,” etc., may be used to describevarious components, such components must not be limited to the aboveterms. The above terms are used only to distinguish one component fromanother.

A main object used herein means an object that a user of an imageacquisition device 1000 desires to include in an image, and a sub-objectused herein means an object that the user does not desire to include inan image.

The disclosure will now be described more fully with reference to theaccompanying drawings, in which exemplary embodiments of the disclosureare shown.

FIG. 1 illustrates an example in which the image acquisition device 1000acquires an image, according to some embodiments.

Referring to FIG. 1, the image acquisition device 1000 may acquire animage by using a camera included in the image acquisition device 1000.In the example of FIG. 1, objects 1 a, 1 b, 2 a, 2 b, and 2 c are in afield of view of the camera. In the example of FIG. 1, object 1 b is abuilding and objects 2 a, 2 b, 2 c and 1 b are people. Portions of thebuilding 1 b are not visible to the camera because they are hiddenbehind objects 2 a, 2 b, 2 c, and 1 a. The image acquired by the imageacquisition device 1000 may include main objects 1 a and 1 b andsub-objects 2 a, 2 b, and 2 c.

When needed for clarity, the three-dimensional object in the field ofview of the camera will be distinguished from an image of thethree-dimensional object captured by the camera. The three-dimensionalthing in the field of view may be referred to directly as the objectwhile the image of the object may be referred to as data correspondingto the object. For example, the building 1 b is an object in the fieldof view of the camera. The portion of an image representing the building1 b may be referred to as data corresponding to the building 1 b. Whenthere is not a risk of confusion when discussing the content of animage, the data representing an object may be referred to as the objectin the image, or the like.

The image acquisition device 1000 may detect the main objects 1 a and 1b and the sub-objects 2 a, 2 b, and 2 c from the acquired image. Theimage acquisition device 1000 may remove the sub-objects 2 a, 2 b, and 2c from the image. In some embodiments, removal of a sub-object may beachieved by replacing the data associated with the sub-object withdefault data representing a blank screen. The removal of the sub-objects2 a, 2 b, and 2 c from the image, in some embodiments, is based on aresult of the detection of the main objects 1 a and 1 b and thesub-objects 2 a, 2 b, and 2 c. The image acquisition device 1000 mayrestore at least a portion of the main object 1 b hidden by thesub-objects 2 a, 2 b, and 2 c, thereby acquiring an image. For example,in the lower portion of FIG. 1 showing the image acquisition device 1000with a second image, after removal of the sub-objects 2 a, 2 b, and 2 c,the building 1 b now appears complete and continuous behind the object 1a. For example, a complete vertical line marking a corner of thebuilding 1 b, formerly partially hidden by the object 2 c, now appearscomplete to a viewer of the image.

The image acquired by the image acquisition device 1000 may include afinal image that is to be stored in the image acquisition device 1000,and a preview image that is displayed on the image acquisition device1000 in order to obtain the final image.

The image acquisition device 1000 may detect at least one of the mainobjects 1 a and 1 b and the sub-objects 2 a, 2 b, and 2 c from theacquired image by using a trained model 3000. The image acquisitiondevice 1000 may restore at least a portion of the main object 1 b hiddenby the sub-objects 2 a, 2 b, and 2 c, by using the trained model 3000.After the restoration of the hidden part of the object 1 b, a cumulativeeffect to a viewer of the image is that of a complete and continuousrepresentation of the object 1 b including the portion that waspreviously hidden by one or more of the sub-objects.

The trained model 3000 may include a plurality of trained models. Inother words, the trained models corresponding to each of the varioususes can be collectively referred to as the trained model 3000. Forexample, the trained model 3000 includes a first trained model fordetecting at least one of the main objects 1 a and 1 b and detecting atleast one of the sub-objects 2 a, 2 b, and 2 c among the plurality ofobjects included in the image and a second trained model for restoringat least a portion of the main object 1 b hidden by the sub-objects 2 a,2 b and 2 c. Hereinafter, various trained models for implementing thedisclosed embodiments are collectively described as a trained model3000.

The trained model 3000 may be established considering, for example, anapplication field of a recognition model, a purpose of learning, or thecomputer performance of a device. The trained model 3000 may include,for example, a model based on an AI neural network. For example, amodel, such as a deep neural network (DNN), a recurrent neural network(RNN), or a bidirectional recurrent DNN (BRDNN), and GenerativeAdversarial Networks (GAN) may be used as the trained model 3000, butembodiments are not limited thereto.

According to an embodiment, the trained model 3000 may learn learningdata according to a preset standard in order to detect at least one ofthe main objects 1 a and 1 b and the sub-objects 2 a, 2 b, and 2 c fromthe acquired image. For example, the trained model 3000 may detect atleast one of the main objects 1 a and 1 b and the sub-objects 2 a, 2 b,and 2 c by performing supervised learning, unsupervised learning, andreinforcement learning with respect to the learning data. The trainedmodel 3000 may detect at least one of the main objects 1 a and 1 b andthe sub-objects 2 a, 2 b, and 2 c from the acquired image by learningthe learning data according to DNN technology.

According to an embodiment, the trained model 3000 may learn thelearning data according to a preset standard in order to restore atleast a portion of the main object 1 b hidden by the sub-objects 2 a, 2b, and 2 c. For example, the trained model 3000 may restore at least aportion of the main object 1 b hidden by the sub-objects 2 a, 2 b, and 2c by performing supervised learning, unsupervised learning, andreinforcement learning with respect to the learning data. The trainedmodel 3000 may restore at least a portion of the main object 1 b hiddenby the sub-objects 2 a, 2 b, and 2 c, by learning the learning dataaccording to DNN technology.

According to an embodiment, the user may acquire at least one of thepreview image and the final image, as an image including only mainobjects.

FIG. 2 is a flowchart of a method, performed by the image acquisitiondevice 1000, of acquiring an image, according to some embodiments. FIGS.3 through 5 are views illustrating examples of displaying an indicatorindicating a sub-object, according to some embodiments.

Referring to FIG. 2, in operation S210 the image acquisition device 1000may acquire a first image by using the camera. In general the cameracaptures light from three-dimensional objects in the field of view ofthe camera and records two-dimensional images consisting of datacorresponding to the three-dimensional objects. In operation S230, theflowchart of FIG. 2 illustrates that the image acquisition device 1000may detect at least one of a main object and a sub-object from the firstimage. Specifically, this may be referred to as detecting first datacorresponding to the main object and detecting second data correspondingto the sub-object. However, generally, the data in the image may bereferred to directly as the object when there is no risk of confusion.In operation S250, the flowchart of FIG. 2 illustrates that the imageacquisition device 1000 may display an indicator indicating the detectedsub-object.

According to an embodiment, the image acquisition device 1000 mayacquire the first image by using the camera included in the imageacquisition device 1000. Alternatively, the image acquisition device1000 may acquire an image from an external camera connected to the imageacquisition device 1000 according to at least one of a wired manner anda wireless manner. The first image acquired by the camera may include apreview image for acquiring a final image that is to be stored in theimage acquisition device 1000. The preview image may be displayed on adisplay of the image acquisition device 1000 or may be displayed on anexternal display connected to the image acquisition device 1000according to at least one of a wired manner and a wireless manner.

In operation S230, the image acquisition device 1000 may detect a mainobject and a sub-object from the first image. The main object mayinclude an object that the user of the image acquisition device 1000desires to include in the final image. The sub-object may also includeanother object that the user does not desire to include in the finalimage.

Referring to FIGS. 3 through 5, the image acquisition device 1000 maydetect the main objects 1 a and 1 b and the sub-objects 2 a, 2 b, and 2c from the first image.

According to an embodiment, the main object 1 a may include a human. Forexample, the main object 1 a may include the user of the imageacquisition device 1000. Alternatively, the main object 1 a may includea person associated with the user of the image acquisition device 1000.In detail, the main object 1 a may include a family member, a lover, ora relative of the user of the image acquisition device 1000.

According to an embodiment, the main object 1 b may include a building,a sculpture, and/or a natural landscape. For example, the main object 1b may include a landmark of a region where the image acquisition device1000 is located. As another example, the main object 1 b may include asculpture located around or near the image acquisition device 1000. Asanother example, the main object 1 b may include natural landscapes,such as the sky, lawn, a lake, and/or the sea.

According to an embodiment, the sub-objects 2 a, 2 b, and 2 c mayinclude people. For example, the sub-objects 2 a, 2 b, and 2 c mayinclude humans located near the user. In detail, the sub-objects 2 a, 2b, and 2 c may include a passerby passing near the user, a person who istaking a picture with the main object 1 b as a background, and a personwho is selling something near the user. In general, the persons includedin the sub-objects 2 a, 2 b, and 2 c may be unrelated to the user, butembodiments are not limited thereto.

According to an embodiment, a sub-object may include a specific object.For example, a sub-object may include at least one object that hides themain object 1 b or a portion of the main object 1 b within the firstimage. For example, when a sub-object in the field of view blocks somelight from the main object 1 b reaching the camera, this blockage oflight means that part of the main object 1 b is hidden from the cameraby the sub-object. Because of the blockage of light, data that ispresent in the image captured by the camera is incomplete in the sensethat portions of the main object 1 b are not visible in the imagecaptured by the camera. As examples, a sub-object may include objects,such as a tree, a trash can, a newsstand, a wall, and a barbed-wirefence, that hide the main object 1 b, but embodiments are not limitedthereto.

According to an embodiment, the main objects 1 a and 1 b and thesub-objects 2 a, 2 b, and 2 c may be detected from the first image byinputting the first image to an AI neural network. For example, the AIneural network may detect various portions of data within the firstimage, the various portions of data corresponding respectively to thedifferent sub-objects. For example, the image acquisition device 1000may input the first image to the AI neural network, and the AI neuralnetwork may detect at least one of the main objects 1 a and 1 b and thesub-objects 2 a, 2 b, and 2 c from the first image, based on a result oflearning the learning data.

According to an embodiment, the AI neural network may learn by using atleast one image stored in the image acquisition device 1000 as thelearning data. For example, the AI neural network may use, as thelearning data, an image including the face of the user and imagesincluding the faces of people associated with the user, wherein theimages are stored in the image acquisition device 1000.

In many cases, the user of the image acquisition device 1000 takespictures including his or her face and the faces of people associatedwith the user. Accordingly, a plurality of images each including theface of the user and a plurality of images each including people (familymembers, a lover, and relatives) associated with the user may be storedin the image acquisition device 1000. Accordingly, when the AI neuralnetwork learns, as the learning data, at least one image stored in theimage acquisition device 1000, the AI neural network may detect at leastone of the user and the people associated with the user, as the mainobject 1 a.

The image acquisition device 1000 may store a plurality of images eachassociated with a specific region where the user is located. When theuser wants to travel to a specific region, the user generally searchesfor materials associated with the region on the Internet, and storesimages associated with the region in the image acquisition device 1000.In particular, the user generally stores images associated with majorbuildings, sculptures, and natural landscapes of the specific region inthe image acquisition device 1000. Accordingly, when the AI neuralnetwork learns, as the learning data, at least one image stored in theimage acquisition device 1000, the AI neural network may detect at leastone of the major buildings, the sculptures, and the natural landscapesof the specific region where the user is located, as the main object 1b.

According to an embodiment, the AI neural network may learn by using atleast one image disclosed on the Internet as the learning data. Forexample, the AI neural network may use an image disclosed on theInternet and associated with the specific region, as the learning data.When the user wants to travel to a specific region, the user may searchfor images associated with the specific region on the Internet. The AIneural network may detect a building, a sculpture, and/or a naturallandscape of the specific region where the user is located, as the mainobject 1 b. The AI neural network may perform the detection by learning,as the learning data, images associated with the specific region foundby the user on the Internet.

The AI neural network may use images disclosed on the Internet andassociated with general objects, as the learning data. The AI neuralnetwork may use a plurality of images disclosed on the Internet, as thelearning data, in order to recognize general objects (e.g., a tree, atrash can, a streetlamp, traffic lights, a stall, a human, and ananimal) within an image. The AI neural network may detect thesub-objects 2 a, 2 b, and 2 c from the first image by learning, as thelearning data, the images associated with the general objects disclosedon the Internet.

In operation S250, the image acquisition device 1000 may display anindicator indicating that a detected object is a sub-object. Forexample, the image acquisition device 1000 may display both the firstimage and an indicator indicating a sub-object, on the display includedin the image acquisition device 1000. Alternatively the imageacquisition device 1000 may display both the first image and anindicator indicating a sub-object on the external display connected tothe image acquisition device 1000, wherein the connection may be a wiredconnection or a wireless connection.

Referring to FIGS. 3 through 5, the image acquisition device 1000 maydisplay an indicator indicating a sub-object according to variousmethods.

According to an embodiment, the image acquisition device 1000 maydisplay indicators 3 a, 3 b, and 3 c near the sub-objects 2 a, 2 b, and2 c, as shown in FIG. 3. In detail, the image acquisition device 1000may display indicators 3 a, 3 b, and 3 c each shaped as a star, near thesub-objects 2 a, 2 b, and 2 c. One of ordinary skill in the art willrecognize that an indicator may be displayed in various shapes.

According to an embodiment, the image acquisition device 1000 maydisplay an indicator indicating a sub-object, by indicating thesub-objects 2 a, 2 b, and 2 c in dashed lines, as shown in FIG. 4. Forexample, the image acquisition device 1000 may display the sub-objects 2a, 2 b, and 2 c by outlining the sub-objects 2 a, 2 b, and 2 c in dashedlines. As another example, the image acquisition device 1000 may displaythe sub-objects 2 a, 2 b, and 2 c in semi-transparent dashed lines suchthat at least a portion of the main object 1 b hidden by the sub-objects2 a, 2 b, and 2 c is displayed within the sub-objects 2 a, 2 b, and 2 c.

According to an embodiment, the image acquisition device 1000 maydisplay an indicator indicating a sub-object by overlaying at leastrespective portions of the sub-objects 2 a, 2 b, and 2 c with at leastone of a preset color and a preset pattern, as shown in FIG. 5.

FIG. 6 is a flowchart of a method of restoring at least a portion of amain object hidden by a sub-object, according to some embodiments. FIGS.7 through 11 illustrate examples of restoring at least a portion of amain object hidden by a sub-object, according to some embodiments.

Referring to FIG. 6, the image acquisition device 1000 may acquire thefirst image by using the camera, in operation S610, detect at least oneof a main object and a sub-object from the first image, in operationS630, remove the detected sub-object from the first image, in operationS650, generate a second image by restoring an area where the removedsub-object was located, in operation S670, and store the generatedsecond image, in operation S690.

Operation S610 is similar to operation S210, and thus a redundantdescription thereof will be omitted.

Operation S630 is similar to operation S230, and thus a redundantdescription thereof will be omitted.

In operation S650, the image acquisition device 1000 may remove thedetected sub-object from the first image. In some embodiments removingthe detected sub-object corresponds to identifying data within the imageto be overwritten by a second image.

According to an embodiment, as shown in FIG. 7, the image acquisitiondevice 1000 may remove, from the first image, data associated with atleast some areas of the first image where the detected sub-objects 2 a,2 b, and 2 c are located. In detail, the image acquisition device 1000may remove only data associated with the sub-objects 2 a, 2 b, and 2 cfrom the first image. Alternatively, as shown in areas 5 a, 5 b, and 5 cof FIG. 8, the image acquisition device 1000 may remove, from the firstimage, data associated with areas 4 a, 4 b, and 4 c including thesub-objects 2 a, 2 b, and 2 c. The image acquisition device 1000 maydisplay or not display an image from which data associated with asub-object has been removed.

According to an embodiment, the image acquisition device 1000 mayremove, from the first image, the data associated with the at least someareas of the first image where the sub-objects 2 a, 2 b, and 2 c arelocated, by using the AI neural network. The AI neural network maydetect an area associated with a sub-object such that restoration isefficiently performed, and remove the detected area, by learning as thelearning data a partially-removed image and an image in which a removedportion has been restored.

In operation S670, the image acquisition device 1000 may restore the atleast some areas of the first image from which data has been removed. Insome embodiments, restoration of removed data corresponds to replacingremoved data with other data, where the other data corresponds to partof an object, for example a part of a building, that was hidden from thecamera at the time the first image was captured.

According to an embodiment, the image acquisition device 1000 mayrestore the areas 5 a, 5 b, and 5 c from which data has been removed,such that at least a portion of the main object 1 b hidden by thesub-objects 2 a, 2 b, and 2 c is included. Referring to FIG. 9, theimage acquisition device 1000 may restore the areas 5 a, 5 b, and 5 cfrom which data has been removed, such that the at least portion of themain object 1 b is included, thereby obtaining restored areas 6 a, 6 b,and 6 c.

According to an embodiment, the image acquisition device 1000 mayrestore an area from which data has been removed, such that the at leastportion of the main object 1 b is included, by using the AI neuralnetwork. For example, the AI neural network may restore an area suchthat the at least portion of the main object 1 b is included, bylearning, as the learning data, at least one image associated with themain object 1 b. In detail, the AI neural network may learn, as thelearning data, at least one image stored in the image acquisition device1000 and associated with the main object 1 b. The AI neural network maylearn at least one image disclosed on the Internet and associated withthe main object 1 b, as the learning data. The AI neural network maylearn images associated with general objects, as the learning data. TheAI neural network may generate a second image by restoring adata-removed area such that the at least portion of the main object 1 bis included, by using a model such as a generative adversarial network(GAN).

According to an embodiment, the image acquisition device 1000 mayrestore the data-removed area such that sharpness of a restored areacorresponds to sharpness of the vicinity of the restored area. Sharpnesscorresponds to a subjective perception that is related to the edgecontrast of an image. The image acquisition device 1000 may acquire afirst image in which the main object 1 a is “in focus” and the mainobject 1 b is “out of focus,” e.g., a first sharpness.” In this case,the main object 1 a in focus may be clear, and the main object 1 b outof focus may be unclear, such as, blurred. The image acquisition device1000 may restore the data-removed area such that the sharpness of therestored area, e.g., a second sharpness, corresponds to sharpness, inthis example, blurry because out of focus, e.g., a first sharpness, ofthe main object 1 b included in the first image.

According to an embodiment, in order to restore the data-removed areasuch that the sharpness of the restored area corresponds to thesharpness of the vicinity of the restored area, the image acquisitiondevice 1000 may determine the sharpness of the main object 1 b includedin the restored area to correspond to the sharpness of the main object 1b included in the first image. The image acquisition device 1000 mayadditionally perform image processing such that the sharpness of themain object 1 b included in the restored area corresponds to thesharpness of the main object 1 b included in the first image.

Referring to FIG. 10, the areas of the main object 1 b included inrestored areas 6 d, 6 e, and 6 f may be clearer than the vicinity of therestored areas 6 d, 6 e, and 6 f. Referring to FIG. 11, the imageacquisition device 1000 may perform image processing such that sharpnessof a clearly restored area as in FIG. 10 corresponds to the sharpness ofthe vicinity of the restored area. For example, the image acquisitiondevice 1000 may perform image processing for lowering the sharpness ofthe main object 1 b included in the restored areas 6 d, 6 e, and 6 f.Sharpness of the main object 1 b included in image-processing-completedareas 6 g, 6 h, and 6 i may correspond to the sharpness of the mainobject 1 b included in the first image.

Although it has been described above that the hidden areas of the mainobject 1 b are restored clearly and then image processing for loweringthe sharpness of the clearly restored areas of the main object 1 b isadditionally performed, embodiments are not limited thereto. Withoutperforming the image processing for lowering the sharpness of theclearly restored areas of the main object 1 b, the hidden areas of themain object 1 b may be restored such that sharpness of the main object 1b included in restored areas corresponds to sharpness of the vicinity ofthe restored areas

FIG. 12 is a flowchart of a method of detecting a sub-object incorrespondence with a user input, according to some embodiments. FIGS.13 and 14 illustrate examples of detecting a sub-object incorrespondence with a user input, according to some embodiments.

Referring to FIG. 12, the image acquisition device 1000 may receive aninput of selecting an object included in a first image, in operationS1210, detect the selected object as a sub-object, in operation S1230,and display an indicator indicating that the detected object is asub-object, in operation S1250.

In operation S1210, the image acquisition device 1000 may receive aninput of selecting an object included in the first image from the uservia a user input interface.

Referring to FIG. 13, a user 10 may select an object 2 c included in thefirst image displayed on a touch screen of the image acquisition device1000. For example, the user 10 may select an area where the object 2 cis located, by using the touch screen.

In operation S1230, the image acquisition device 1000 may detect theselected object 2 c as a sub-object.

According to an embodiment, the image acquisition device 1000 may detectthe selected object 2 c as a sub-object by using the AI neural network.The AI neural network may detect the appearances of objects (e.g., atree, a trash can, a streetlamp, traffic lights, a stall, a human, andan animal) included in an image by learning images of the appearances ofobjects as the learning data. The image acquisition device 1000 maydetect an object that includes an area that has received a user input oris located in the vicinity of the area, by using an AI neural network.

Referring to FIG. 13, the image acquisition device 1000 may detect theobject 2 c located in the vicinity of an area that has received an inputof the user 10. The image acquisition device 1000 may detect theappearance of the object 2 c, which is also referred to as a human 2 c,located in the vicinity of the area that has received an input of theuser 10, by using the AI neural network. The image acquisition device1000 may detect the human 2 c as a sub-object, based on the appearanceof the detected human 2 c.

In operation S1250, the image acquisition device 1000 may display anindicator indicating that the selected object 2 c is a sub-object.

Referring to FIG. 14, the image acquisition device 1000 may display anindicator in a star shape indicating that the selected object 2 c is asub-object, near the object 2 c. However, embodiments are not limitedthereto. As shown in FIG. 4, the object 2 c may be marked by a dashedline to indicate that the object 2 c is a sub-object. As shown in FIG.5, at least a portion of the object 2 c may be overlaid with at leastone of a preset color and a preset pattern and then displayed. The imageacquisition device 1000 may display an indicator indicating that theselected object 2 c is a sub-object, based on the appearance of theobject 2 c detected using the AI neural network.

FIG. 15 is a flowchart of a method of restoring at least a portion of amain object hidden by a sub-object in correspondence with a user input,according to some embodiments. FIGS. 16 through 18 illustrate examplesof restoring at least a portion of a main object hidden by a sub-object,in correspondence with a user input, according to some embodiments.

Referring to FIG. 15, the image acquisition device 1000 may receive aninput of selecting an object included in a first image (a “selection” ofthe representation, data, or appearance of the object in the firstimage), in operation S1510, detect the selected object as a sub-object,in operation S1530, remove the selected object from the first image, inoperation S1550, and generate a second image by restoring an area wherethe removed object was located, in operation S1570.

In operation S1510, the image acquisition device 1000 may receive aninput of selecting an object included in the first image, from the uservia the user input interface.

Referring to FIG. 16, the user 10 may select the object 2 c included inthe first image displayed on the touch screen of the image acquisitiondevice 1000. For example, the user 10 may select an area where theobject 2 c is located, by using the touch screen.

In operation S1530, the image acquisition device 1000 may detect theselected object 2 c as a sub-object. Operation S1530 is similar tooperation S1230, and thus a redundant description thereof will beomitted.

In operation S1550, the image acquisition device 1000 may remove theselected object 2 c from the first image.

According to an embodiment, the image acquisition device 1000 mayremove, from the first image, data associated with at least some area ofthe first image where the object 2 c is located. For example, the imageacquisition device 1000 may remove only data associated with thesub-object 2 c from the first image. Alternatively, as shown in an area7 a of FIG. 17, the image acquisition device 1000 may remove, from thefirst image, data associated with the area 7 a including the sub-object2 c. Operation S1550 is similar to operation S650, and thus a redundantdescription thereof will be omitted.

In operation S1570, the image acquisition device 1000 may restore atleast a portion of the first image from which data has been removed.

According to an embodiment, the image acquisition device 1000 mayrestore the data-removed area 7 a such that at least a portion of themain object 1 b hidden by the sub-object 2 c is included. Referring toFIG. 18, the image acquisition device 1000 may restore the data-removedarea 7 a such that at least a portion of the main object 1 b isincluded, thereby generating a restored area 7 b. Operation S1570 issimilar to operation S670, and thus a redundant description thereof willbe omitted.

FIG. 19 is a flowchart of a method of tracking a motion of a sub-objectwithin an image, according to some embodiments. FIGS. 20 through 23 areviews illustrating examples of tracking a motion of a sub-object withinan image, according to some embodiments.

Referring to FIG. 19, the image acquisition device 1000 may acquire afirst image by using the camera, in operation S1910, detect at least oneof a main object and a sub-object from the first image, in operationS1930, track a motion of the detected sub-object, in operation S1950,and display an indicator indicating the detected sub-object, based onthe motion of the sub-object, in operation S1970.

Operation S1910 is similar to operation S210, and thus a redundantdescription thereof will be omitted.

Operation S1930 is similar to operation S230, and thus a redundantdescription thereof will be omitted.

In operation S1950, the image acquisition device 1000 may track themotion of the detected sub-object.

According to an embodiment, the image acquisition device 1000 may trackmotions of the sub-objects 2 a and 2 b by inputting the first image tothe AI neural network. Referring to FIG. 20, the AI neural network maydetect the appearances of the sub-objects 2 a and 2 b from the firstimage and may track the motions of the sub-objects 2 a and 2 b, based onthe detected appearances of the sub-objects 2 a and 2 b. For example,the AI neural network may track the motion of the sub-object 2 a fromthe left to the right by recognizing the appearance of the sub-object 2a as the shape of a human who walks from the left to the right. Asanother example, the AI neural network may track the motion of thesub-object 2 b from a left lower end to a right upper end by recognizingthe appearance of the sub-object 2 b as the shape of a human who walksfrom the left lower end to the right upper end.

According to an embodiment, the image acquisition device 1000 may trackthe motions of the sub-objects 2 a and 2 b by using a sensor included inthe camera. The sensor included in the camera may include a sensorcapable of detecting a phase difference of an object. The phasedifference detection sensor may detect a phase change caused accordingto the motions of the sub-objects 2 a and 2 b. The image acquisitiondevice 1000 may track the motions of the sub-objects 2 a and 2 b, basedon changes in the phases of the detected sub-objects 2 a and 2 b.

According to an embodiment, the image acquisition device 1000 mayacquire a plurality of images by using the camera and track the motionof the detected sub-object, based on the acquired plurality of images.The image acquisition device 1000 may track the motions of thesub-objects 2 a and 2 b by inputting the acquired plurality of images tothe AI neural network. For example, the image acquisition device 1000may track the motions of the sub-objects 2 a and 2 b by detecting motionvectors of the sub-objects 2 a and 2 b from the first image. The imageacquisition device 1000 may detect the motion vectors of the sub-objects2 a and 2 b by acquiring a plurality of images before and after thefirst image by using the camera and inputting the plurality of imagesacquired before and after the first image to the AI neural network. Forexample, the AI neural network may detect the motion vectors of thesub-objects 2 a and 2 b by comparing the sub-objects 2 a and 2 bincluded in the first image with sub-objects 2 a and 2 b included in theplurality of images acquired before and after the first image. Asanother example, the AI neural network may detect the motion vectors ofthe sub-objects 2 a and 2 b by splitting the first image into aplurality of blocks and each of the plurality of images acquired beforeand after the first image into a plurality of blocks and comparing theplurality of blocks of the first image with the plurality of blocks ofeach of the plurality of images acquired before and after the firstimage.

In operation S1970, the image acquisition device 1000 may display anindicator indicating the detected sub-object, based on the motion of thedetected sub-object.

Referring to FIGS. 20 and 21, the image acquisition device 1000 maydisplay the indicators 3 a and 3 b indicating the sub-objects 2 a and 2b to correspond to the motions of the sub-objects 2 a and 2 b. As thesub-object 2 a moves to the right, the image acquisition device 1000 maydisplay the indicator 3 a such that the indicator 3 a located near thesub-object 2 a is located near the sub-object 2 a moved to the right. Asthe sub-object 2 b moves to the right upper end, the image acquisitiondevice 1000 may display the indicator 3 b such that the indicator 3 blocated near the sub-object 2 b is located near the sub-object 2 b movedto the right upper end.

Referring to FIGS. 22 and 23, the image acquisition device 1000 maydisplay the appearances of the sub-objects 2 a and 2 b in dashed lines.The image acquisition device 1000 may display the appearances of thesub-objects 2 a and 2 b in dashed lines, to correspond to the motions ofthe sub-objects 2 a and 2 b.

When the image acquisition device 1000 overlays at least respectiveportions of the sub-objects 2 a and 2 b with at least one of a presetcolor and a preset pattern and displays a result of the overlaying, theimage acquisition device 1000 may overlay at least respective portionsof the moving sub-objects 2 a and 2 b with at least one of a presetcolor and a preset pattern to correspond to the motions of thesub-objects 2 a and 2 b, and may display a result of the overlaying.

FIG. 24 is a flowchart of a method of restoring at least a portion of amain object hidden by a sub-object, based on relative locations betweena camera and at least one of the main object and the sub-object,according to some embodiments. FIGS. 25 and 26 illustrate examples ofdetecting a main object and a sub-object when the camera moves,according to some embodiments.

Referring to FIG. 24, the image acquisition device 1000 may acquire afirst image by using the camera, in operation S2410, detect the mainobject and the sub-object from the first image, in operation S2430,acquire information about the relative locations between the camera andat least one of the main object and the sub-object, in operation S2450,and display a second image, based on the acquired information about therelative locations, in operation S2470.

Operation S2410 is similar to operation S210, and thus a redundantdescription thereof will be omitted.

Operation S2430 is similar to operation S230, and thus a redundantdescription thereof will be omitted.

In operation S2450, the image acquisition device 1000 may acquire theinformation about the relative locations between the camera and at leastone of the main object and the sub-object.

According to an embodiment, the image acquisition device 1000 mayacquire location information of the camera. For example, the imageacquisition device 1000 may acquire the location information of thecamera by using a position sensor, such as a global positioning system(GPS) module. As another example, the image acquisition device 1000 mayacquire the location information of the camera by using a short-rangecommunication technique, such as Wifi, Bluetooth, Zigbee, and beacon.

According to an embodiment, the image acquisition device 1000 mayacquire the location information of the camera from the first image byusing the AI neural network.

For example, the AI neural network may recognize the main object 1 bincluded in the first image and acquire location information of a regionwhere the recognized object 1 b is located. In detail, when the mainobject 1 b is a major building in a certain region, the AI neuralnetwork may acquire location information of the region where the mainobject 1 b is located. Because the camera that captures the first imageincluding the main object 1 b will be located in a region similar to theregion where the main object 1 b is located, the AI neural network mayacquire the location information of the camera.

As another example, the AI neural network may acquire the locationinformation of the main object 1 b by using the location information ofthe region where the main object 1 b included in the first image islocated, photographing information (e.g., a focal length) of the firstimage, and a size of the main object 1 b within the first image.

According to an embodiment, the image acquisition device 1000 mayacquire location information of a main object. For example, the imageacquisition device 1000 may acquire the location information of thecamera by using the position sensor, such as a GPS module, and mayacquire location information of the main object by using thephotographing information (e.g., a focal length) of the first image, anda size of the main object within the first image.

According to an embodiment, the image acquisition device 1000 mayacquire the location information of the main object from the first imageby using the AI neural network. For example, the AI neural network mayrecognize the main object 1 b included in the first image and acquirelocation information of a region where the recognized main object 1 b islocated. In detail, when the main object 1 b is a major building in acertain region, the AI neural network may acquire location informationof the region where the main object 1 b is located.

According to an embodiment, the image acquisition device 1000 mayacquire pieces of location information of the sub-objects 2 a, 2 b, and2 c. For example, the image acquisition device 1000 may acquire thelocation information of the camera by using the position sensor, such asa GPS module, and may acquire the location information of the mainobject by using the photographing information (e.g., a focal length) ofthe first image and respective sizes of the sub-objects 2 a, 2 b, and 2c within the first image.

According to an embodiment, the image acquisition device 1000 mayacquire information of relative locations between the camera and themain object. For example, the image acquisition device 1000 may acquirethe information of the relative locations between the camera and themain object, based on the acquired location information of the cameraand the acquired location information of the main object. Theinformation of relative locations depends, for example, on depth ordistance from the camera to an object in the field of view of thecamera. As another example, the image acquisition device 1000 mayacquire the information of the relative locations between the camera andthe main object by using the photographing information (e.g., a focallength) of the first image and the size of the main object within thefirst image, by using the AI neural network.

According to an embodiment, the image acquisition device 1000 mayacquire information of relative locations between the camera and thesub-object. For example, the image acquisition device 1000 may acquirethe information of the relative locations between the camera and thesub-object, based on the acquired location information of the camera andlocation information of the sub-object. As another example, the imageacquisition device 1000 may acquire the information of the relativelocations between the camera and the sub-object by using thephotographing information (e.g., a focal length) of the first image andthe size of the sub-object within the first image, by using the AIneural network.

In operation S2470, the image acquisition device 1000 may generate asecond image, based on the information of the relative locations betweenthe camera and at least one of the main object and the sub-object.

According to an embodiment, the image acquisition device 1000 mayremove, from the first image, data associated with at least some areasof the first image where the sub-objects 2 a, 2 b, and 2 c arerespectively located, by using the AI neural network.

For example, the AI neural network may track the motions of thesub-objects 2 a, 2 b, and 2 c, based on information of relativelocations between the camera and the sub-objects 2 a, 2 b, and 2 c. Forexample, the AI neural network may detect a change in the information ofthe relative locations between the camera and the sub-objects 2 a, 2 b,and 2 c, by using the phase difference detection sensor included in thecamera. For example, relative locations may be determined both at afirst time and at a second time. The change in information is the changein the relative location information from the first time to the secondtime. As another example, the AI neural network may detect the change inthe information of the relative locations between the camera and thesub-objects 2 a, 2 b, and 2 c, by comparing a plurality of images witheach other. As another example, the AI neural network may detect thechange in the information of the relative locations between the cameraand the sub-objects 2 a, 2 b, and 2 c, by detecting motion vectors froma plurality of images acquired using the camera. The AI neural networkmay predict directions and speeds of the motions of the sub-objects 2 a,2 b, and 2 c, based on the change in the information of the relativelocations between the camera and the sub-objects 2 a, 2 b, and 2 c. TheAI neural network may track the motions of the sub-objects 2 a, 2 b, and2 c, based on the predicted directions and speeds of the motions of thesub-objects 2 a, 2 b, and 2 c. Tracking a motion of a sub-object byusing the AI neural network is similar to operation S1950, and thus aredundant description thereof will be omitted.

Referring to FIGS. 25 and 26, as the camera becomes farther from themain objects 1 a and 1 b and the sub-objects 2 a, 2 b, and 2 c,information of relative locations between the camera and the mainobjects 1 a and 1 b and the sub-objects 2 a, 2 b, and 2 c may change.When the user takes a picture, the user may change a location of thecamera to change the composition of the picture. For example, the usermay take a picture by moving one step backwards from the main object 1b, which is also referred to as a building 1 b, of a certain region, inorder to capture in a photograph a wider view of the building 1 b.Accordingly, information of relative locations between the camera andeach of the main objects 1 a and 1 b and the sub-objects 2 a, 2 b, and 2c may be change because the user with the camera moved a furtherdistance away from the building 1 b.

The image acquisition device 1000 may track the motions of thesub-objects 2 a, 2 b, and 2 c, based on the changed locationinformation. The image acquisition device 1000 may display an indicatorindicating the sub-objects 2 a and 2 b in correspondence with themotions of the sub-objects 2 a and 2 b. Referring to FIGS. 25 and 26,the image acquisition device 1000 may track motions of the sub-objects 2a and 2 b in a direction toward a right upper end. The image acquisitiondevice 1000 may display the indicator indicating the sub-objects 2 a and2 b by marking or rendering the appearances of the sub-objects 2 a and 2b in dashed lines in correspondence with the motions of the sub-objects2 a and 2 b. In some embodiments, data in the image corresponding to anobject in the field of view of the camera may be referred to as anappearance corresponding to the object, or simply as an appearance.

The AI neural network may efficiently remove, from the first image, dataassociated with at least some areas of the first image where thesub-objects 2 a, 2 b, and 2 c are respectively located, based on themotions of the sub-objects 2 a, 2 b, and 2 c. The removing, by the imageacquisition device 1000, from the first image, the data associated withat least some areas of the first image where the sub-objects 2 a, 2 b,and 2 c are respectively located is similar to operation S650, and thusa redundant description thereof will be omitted.

According to an embodiment, the image acquisition device 1000 may trackthe motions of the sub-objects 2 a, 2 b, and 2 and may remove, from thefirst image, data associated with at least some areas of the first imagewhere the sub-objects 2 a, 2 b, and 2 c are respectively located. Thetracking, by the image acquisition device 1000, of the motions of thesub-objects 2 a, 2 b, and 2 c is similar to operation S1950 and theimage acquisition device 1000 removing, from the first image, the dataassociated with at least some areas of the first image where thesub-objects 2 a, 2 b, and 2 c are respectively located are similar tooperation S650, and thus redundant descriptions thereof will be omitted.

According to an embodiment, the image acquisition device 1000 mayrestore a data-removed area such that at least a portion of the mainobject 1 b hidden by the sub-objects 2 a, 2 b, and 2 c is included,based on the information of the relative locations between the cameraand the main object 1 b, by using the AI neural network. For example,the AI neural network may acquire location information of the mainobject 1 b, which is a building located in a certain region. As inoperation S2450, the AI neural network may acquire information ofrelative locations between the camera and the main object 1 b. The AIneural network may search for an image of the main object 1 b, similarto the first image, based on the location information of the main object1 b and the information of the relative locations between the camera andthe main object 1 b. The AI neural network may restore the data-removedarea of the first image by learning at least one found image as thelearning data. The AI neural network may generate the second image byrestoring the data-removed area within the first image.

According to an embodiment, the image acquisition device 1000 maydisplay the second image on the display by restoring the data-removedarea within the first image. The image acquisition device 1000 may storethe second image in a memory. In this case, the image acquisition device1000 may store the second image in the memory when the second image isgenerated by restoring the data-removed area within the first image, ormay store the second image in the memory in response to a user input.

FIG. 27 is a view illustrating a method of displaying an indicatorindicating a sub-object by using a server 2000, according to someembodiments. FIGS. 28 and 29 are views illustrating examples ofdisplaying an indicator indicating a sub-object by using the server2000, according to some embodiments.

Referring to FIG. 27, the image acquisition device 1000 may receive aninput of driving the camera from the user, in operation S2710, accessthe server 2000, in operation S2720, acquire the first image by usingthe camera, in operation S2730, and transmit the first image to theserver 2000, in operation S2740. The server 2000 may detect a mainobject and a sub-object from the received first image, in operationS2750, and transmit information about a result of the detection of themain object and the sub-object to the image registration device 1000, inoperation S2760. In operation S2770, the image acquisition device 1000may display the indicator indicating the sub-object, based on theinformation received from the server 2000.

In operation S2710, the image acquisition device 1000 may receive aninput of driving the camera, from the user. Referring to FIG. 28, theimage acquisition device 1000 may receive, from the user 10, an inputwith respect to a button for driving the camera. Alternatively, theimage acquisition device 1000 may receive, from the user 10, an inputfor driving an application regarding the camera.

In operation S2720, the image acquisition device 1000 may access theserver 2000.

According to an embodiment, when the image acquisition device 1000receives, from the user, an input of driving the camera, the imageacquisition device 1000 may automatically access the server 2000 byforming a communication link with the server. The communication link maybe, for example, an Internet-based session. The server 2000 may includean AI neural network that detects at least one of the main objects 1 aand 1 b and the sub-objects 2 a, 2 b, and 2 c from the first image. Theserver 2000 may include an AI neural network that removes, from thefirst image, data associated with at least some areas of the first imagewhere the sub-objects 2 a, 2 b, and 2 c are respectively located, andrestores a data-removed area such that at least a portion of the mainobject 1 b hidden by the sub-objects 2 a, 2 b, and 2 c is included.

In operation S2730, the image acquisition device 1000 may acquire thefirst image by using the camera. Operation S2730 is similar to operationS210, and thus a redundant description thereof will be omitted.

In operation S2740, the image acquisition device 1000 may transmit, overthe communication link, the acquired first image to the server 2000.

In operation S2750, the server 2000 may detect the main object and thesub-object from the received first image by using the AI neural network.Referring to FIGS. 28 and 29, the server 2000 may detect the mainobjects 1 a and 1 b and the sub-objects 2 a, 2 b, and 2 c from the firstimage. The AI neural network may be specialized for the user of theimage acquisition device 1000. For example, the AI neural network mayhave learned data associated with the user (e.g., at least one imagestored in the image acquisition device 1000 of the user) as the learningdata. The AI neural network detecting the main object and the sub-objectfrom the first image is similar to operation S230, and thus a redundantdescription thereof will be omitted.

In operation S2760, the server 2000 may transmit to the imageacquisition device 1000 over the communication link a result of thedetection of the main object and the sub-object from the first image byusing the AI neural network. For example, the server 2000 may transmitto the image acquisition device 1000 a first image including indicatorsindicating the main object and the sub-object. Alternatively, the server2000 may transmit, to the image acquisition device 1000, informationindicating objects included in the first image and informationindicating whether each of the objects is a main object or a sub-object.

In operation S2770, the image acquisition device 1000 may displayindicators 3 a, 3 b, and 3 c respectively indicating the sub-objects 2a, 2 b, and 2 c. Operation S2770 is similar to operation S250, and thusa redundant description thereof will be omitted.

When the image acquisition device 1000 accesses the server 2000 inresponse to reception of the input for driving the camera from the user,the image acquisition device 1000 may receive data about the AI neuralnetwork from the server 2000. For example, the image acquisition device1000 may receive, from the server 2000, a software module thatimplements the AI neural network. As another example, the imageacquisition device 1000 may receive, from the server 2000, data used toupdate the AI neural network. The image acquisition device 1000 maydetect the main object and the sub-object from the first image, based onthe received data about the AI neural network.

FIGS. 30 and 31 are block diagrams of the image acquisition device 1000according to some embodiments.

Referring to FIG. 30, the image acquisition device 1000 may include auser input interface 1100, a display 1210, a processor 1300, and acommunication interface 1500. All of the components illustrated in FIG.30 are not essential components of the image acquisition device 1000.More or less components than those illustrated in FIG. 30 may constitutethe image acquisition device 1000.

For example, referring to FIG. 31, the image acquisition device 1000 mayfurther include a sensing unit 1400, an audio/video (AN) input interface1600, an output interface 1200 and a memory 1700, in addition to theuser input interface 1100, the processor 1300, and the communicationinterface 1500.

The user input interface 1100 denotes a unit via which a user inputsdata for controlling the image acquisition device 1000. For example, theuser input interface 1100 may be, but is not limited to, a key pad, adome switch, a touch pad (e.g., a capacitive overlay type, a resistiveoverlay type, an infrared beam type, an integral strain gauge type, asurface acoustic wave type, a piezo electric type, or the like), a jogwheel, or a jog switch. The user input interface 1100 may include atouch screen that receives a touch input of the user, by combining atouch layer with the display 1210.

The user input interface 1100 may receive a user input of selecting atleast one object (a “selection”) displayed on the display 1210.

The output interface 1200 may output an audio signal, a video signal, ora vibration signal, and may include the display 1210, an audio outputinterface 1220, and a vibration motor 1230.

The display 1210 may display information that is processed by the imageacquisition device 1000. For example, the display 1210 may display afirst image and an indicator indicating a sub-object. The display 1210may display a second image obtained by restoring a data-removed areasuch that at least a portion of a main object is included.

The audio output interface 1220 outputs audio data that is received fromthe communication interface 1500 or stored in the memory 1700. The audiooutput interface 1220 also outputs an audio signal (for example, a callsignal receiving sound, a message receiving sound, or a notificationsound) related with a function of the image acquisition device 1000.

The processor 1300 typically controls an overall operation of the imageacquisition device 1000. For example, the processor 1300 may control theuser input interface 1100, the output interface 1200, the sensing unit1400, the communication interface 1500, the NV input interface 1600, andthe like by executing programs stored in the memory 1700. The processor1300 may perform a function of the image acquisition device 1000 ofFIGS. 1 through 29, by executing the programs stored in the memory 1700.

In detail, the processor 1300 may control the user input interface 1100to receive a user input of selecting at least one of the displayedobjects. The processor 1300 may control a microphone 1620 to receive avoice input of the user. The processor 1300 may execute an applicationthat performs an operation of the image acquisition device 1000, basedon a user input, and may control the user input interface to receive theuser input through the executed application. For example, the processor1300 may execute a voice assistant application and may control the NVinput interface 1600 to receive a voice input of the user through themicrophone 1620 by controlling the executed voice assistant application.

The processor 1300 may control the output interface 1200 and the memory1700 of the image acquisition device 1000 to display the first image,the indicator indicating the sub-object, and the second image. Theprocessor 1300 may include at least one micro-chip.

The processor 1300 may detect the main object and the sub-object fromthe first image. The processor 1300 may detect at least one of the mainobject and the sub-object by using the AI neural network.

The processor 1300 may remove the sub-object from the first image andgenerate the second image by restoring an area of the first image fromwhich the sub-object has been removed. The processor 1300 may remove thesub-object from the first image by using the AI neural network andrestore an area of the first image from which the sub-object has beenremoved. In this case, the AI neural network may restore at least aportion of the main object hidden by the sub-object that is removed.

The processor 1300 may acquire information of relative locations betweenthe camera and the main object and may restore the at least portion ofthe main object, based on the information of the relative locations.

The processor 1300 may perform image processing for restoring the atleast portion of the main object, such that sharpness of a restoredportion of the main object corresponds to sharpness of the main objectwithin the first image. For example, the processor 1300 may determinethe sharpness of at least a portion of the main object that is restoredto correspond to the sharpness of the main object within the firstimage, and may perform image processing such that the sharpness of therestored portion of the main object corresponds to the determinedsharpness.

The processor 1300 may detect a sub-object selected from the firstimage, based on a user input. For example, the processor 1300 may detectthe appearance of each of a plurality of objects included in the firstimage by using the AI neural network, and may detect, as a sub-object,an object located in the vicinity of an area that has received a userinput.

The processor 1300 may track a motion of the sub-object within the firstimage. For example, the processor 1300 may acquire information ofrelative locations between the camera and the sub-object and may trackthe motion of the sub-object by detecting a change in the information ofthe relative locations.

The sensing unit 1400 may sense a state of the image acquisition device1000 or a state of the surroundings of the image acquisition device 1000and may transmit information corresponding to the sensed state to theprocessor 1300.

The sensing unit 1400 may include, but is not limited thereto, at leastone selected from a magnetic sensor 1410, an acceleration sensor 1420, atemperature/humidity sensor 1430, an infrared sensor 1440, a gyroscopesensor 1450, a position sensor (e.g., a GPS) 1460, a pressure sensor1470, a proximity sensor 1480, and an RGB sensor 1490 (i.e., anilluminance sensor). Functions of most of the sensors would beinstinctively understood by one of ordinary skill in the art in view oftheir names and thus detailed descriptions thereof will be omittedherein.

The communication interface 1500 may include at least one component thatenables the image acquisition device 1000 to communicate with otherdevices (not shown) and the server 2000. The other device may be acomputing device, such as the image acquisition device 1000, or asensing device, and embodiments are not limited thereto. For example,the communication interface 1500 may include a short-range wirelesscommunication interface 1510, a mobile communication interface 1520, anda broadcasting receiver 1530.

Examples of the short-range wireless communication interface 1510 mayinclude, but are not limited to, a Bluetooth communication interface, aBluetooth Low Energy (BLE) communication interface, a near fieldcommunication (NFC) interface, a wireless local area network (WLAN)(e.g., Wi-Fi) communication interface, a ZigBee communication interface,an infrared Data Association (IrDA) communication interface, a Wi-Fidirect (WFD) communication interface, an ultra wideband (UWB)communication interface, and an Ant+ communication interface.

The mobile communication interface 1520 may exchange a wireless signalwith at least one selected from a base station, an external terminal,and a server on a mobile communication network. Here, examples of thewireless signal may include a voice call signal, a video call signal,and various types of data according to text/multimedia messagestransmission.

The broadcasting receiver 1530 receives a broadcasting signal and/orbroadcasting-related information from an external source via abroadcasting channel. The broadcasting channel may be a satellitechannel, a ground wave channel, or the like. According to embodiments,the image acquisition device 1000 may not include the broadcastingreceiver 1530.

According to an embodiment, the communication interface 1500 maytransmit the first image to the server 2000.

According to an embodiment, the communication interface 1500 may receiveinformation about a result of the detection of the main object and thesub-object from the server 2000.

According to an embodiment, the communication interface 1500 mayreceive, from the server 2000, a software module that implements the AIneural network.

According to an embodiment, the communication interface 1500 mayreceive, from the server 2000, data used to update the AI neuralnetwork.

The AN input interface 1600 inputs an audio signal or a video signal,and may include a camera 1610 and the microphone 1620. The camera 1610may acquire an image frame, such as a still image or a moving picture,via an image sensor in a video call mode or a photography mode. An imagecaptured via the image sensor may be processed by the processor 1300 ora separate image processor (not shown).

The microphone 1620 receives an external audio signal and converts theexternal audio signal into electrical audio data. For example, themicrophone 1620 may receive an audio signal from an external device or auser. The microphone 1620 may receive a voice input of the user. Themicrophone 1620 may use various noise removal algorithms in order toremove noise that is generated while receiving the external audiosignal.

The memory 1700 may store a program used by the processor 1300 toperform processing and control, and may also store data that is input toor output from the image acquisition device 1000.

The memory 1700 may include at least one type of storage medium selectedfrom among a flash memory type, a hard disk type, a multimedia cardmicro type, a card type memory (for example, a secure digital (SD) orextreme digital (XD) memory), a random access memory (RAM), a staticrandom access memory (SRAM), a read-only memory (ROM), an electricallyerasable programmable ROM (EEPROM), a programmable ROM (PROM), magneticmemory, a magnetic disk, and an optical disk.

The programs stored in the memory 1700 may be classified into aplurality of modules according to their functions, for example, a userinterface (UI) module 1710, a touch screen module 1720, and anotification module 1730.

The UI module 1710 may provide a UI, graphical user interface (GUI), orthe like that is specialized for each application and interoperates withthe image acquisition device 1000. The touch screen module 1720 maydetect a touch gesture on a touch screen of a user and transmitinformation regarding the touch gesture to the processor 1300. The touchscreen module 1720 according to an embodiment may recognize and analyzea touch code. The touch screen module 1720 may be configured by separatehardware including a controller.

The notification module 1730 may generate a signal for notifying that anevent has been generated in the image acquisition device 1000. Examplesof the event generated in the image acquisition device 1000 may includecall signal receiving, message receiving, a key signal input, schedulenotification, and the like. The notification module 1730 may output anotification signal in the form of a video signal via the display 1210,in the form of an audio signal via the audio output interface 1220, orin the form of a vibration signal via the vibration motor 1230.

FIG. 32 is a block diagram of the server 2000 according to someembodiments.

Referring to FIG. 32, the server 2000 may include a communicationinterface 2500, a database (DB) 2700, and a processor 2300.

The communication interface 2500 may include at least one component thatenables the serve 2000 to communicate with the image acquisition device1000.

The communication interface 2500 may receive or transmit an image fromor to the image acquisition device 1000.

The DB 2700 may store a trained model and learning data that is appliedin the trained model.

The processor 2300 typically controls an overall operation of the server2000. For example, the processor 2300 may control the DB 2700 and thecommunication interface 2500 by executing the programs stored in the DB2700 of the server 2000. The processor 2300 may perform some of theoperations of the image acquisition device 1000 of FIGS. 1 through 29,by executing the programs stored in the DB 2700.

The processor 2300 may perform at least one of a function of detectingat least one of a main object and a sub-object from a first image, afunction of removing, from the first image, data associated with atleast some area of the first image where the sub-object is located, anda function of restoring the data-removed area such that at least aportion of the main object hidden by the sub-object is included.

The processor 2300 may manage at least one of data necessary fordetecting at least one of the main object and the sub-object from thefirst image, data necessary for removing, from the first image, the dataassociated with at least some area of the first image where thesub-object is located, and data necessary for restoring the data-removedarea such that at least a portion of the main object hidden by thesub-object is included.

FIG. 33 is a block diagram of the processor 1300 according to someembodiments.

Referring to FIG. 33, the processor 1300 may include a data learner 1310and a data recognizer 1320.

The data learner 1310 may learn a standard for detecting the main objectand the sub-object from the first image. The data learner 1310 may learna standard about which data is to be used in order to detect the mainobject and the sub-object from the first image. The data learner 1310may learn the standard for detecting the main object and the sub-objectfrom the first image, by obtaining data for use in learning and applyingthe obtained data to a data recognition model which will be describedlater.

The data learner 1310 may learn a standard for removing, from the firstimage, the data associated with at least some area of the first imagewhere the sub-object is located. The data learner 1310 may learn astandard about which data is to be used in order to remove, from thefirst image, the data associated with at least some area of the firstimage where the sub-object is located. The data learner 1310 may learnthe standard for removing, from the first image, the data associatedwith at least some area of the first image where the sub-object islocated, by obtaining data for use in learning and applying the obtaineddata to the data recognition model.

The data learner 1310 may learn a standard for restoring thedata-removed area such that at least a portion of the main object hiddenby the sub-object is included. The data learner 1310 may learn astandard about which data is to be used in order to restore thedata-removed area such that at least a portion of the main object hiddenby the sub-object is included. The data learner 1310 may learn thestandard for restoring the data-removed area such that at least aportion of the main object hidden by the sub-object is included, byobtaining data for use in learning and applying the obtained data to thedata recognition model.

The data recognizer 1320 may detect the main object and the sub-objectfrom the first image, based on the data. The data recognizer 1320 maydetect the main object and the sub-object from the first image, based oncertain data, by using the trained data recognition model. The datarecognizer 1320 may detect the main object and the sub-object from thefirst image by obtaining the certain data according to a standardpreviously set due to learning and by using the data recognition modelby using the obtained data as an input value. A result value output bythe data recognition model by using the obtained data as an input valuemay be used to update the data recognition model.

The data recognizer 1320 may remove, from the first image, the dataassociated with at least some area of the first image where thesub-object is located, based on the data. The data recognizer 1320 mayremove, from the first image, the data associated with at least somearea of the first image where the sub-object is located, by using thetrained data recognition model. The data recognizer 1320 may remove,from the first image, the data associated with at least some area of thefirst image where the sub-object is located, by obtaining certain dataaccording to a standard previously set due to learning and by using thedata recognition model by using the obtained data as an input value. Aresult value output by the data recognition model by using the obtaineddata as an input value may be used to update the data recognition model.

The data recognizer 1320 may restore the data-removed area such that atleast a portion of the main object hidden by the sub-object is included.based on the data. The data recognizer 1320 may restore the data-removedarea such that at least a portion of the main object hidden by thesub-object is included. by using the trained data recognition model. Thedata recognizer 1320 may restore the data-removed area such that atleast a portion of the main object hidden by the sub-object is included,by obtaining certain data according to a standard previously set due tolearning and by using the data recognition model by using the obtaineddata as an input value. A result value output by the data recognitionmodel by using the obtained data as an input value may be used to updatethe data recognition model.

At least one of the data learner 1310 and the data recognizer 1320 maybe manufactured in the form of at least one hardware chip and may bemounted on a device. For example, at least one of the data learner 1310and the data recognizer 1320 may be manufactured in the form of adedicated hardware chip for AI, or may be manufactured as a portion ofan existing general-purpose processor (for example, a central processingunit (CPU) or an application processor (AP)) or a processor dedicated tographics (for example, a graphics processing unit (GPU)) and may bemounted on any of the aforementioned various devices.

In this case, the data learner 1310 and the data recognizer 1320 may beboth mounted on a single device, or may be respectively mounted onindependent devices. For example, one of the data learner 1310 and thedata recognizer 1320 may be included in the image acquisition device1000, and the other may be included in the server 2000. The data learner1310 and the data recognizer 1320 may be connected to each other by wireor wirelessly, and thus model information established by the datalearner 1310 may be provided to the data recognizer 1320 and data inputto the data recognizer 1320 may be provided as additional learning datato the data learner 1310.

At least one of the data learner 1310 and the data recognizer 1320 maybe implemented as a software module. When at least one of the datalearner 1310 and the data recognizer 1320 is implemented using asoftware module (or a program module including instructions), thesoftware module may be stored in non-transitory computer readable media.In this case, the at least one software module may be provided by anoperating system (OS) or by a certain application. Alternatively, someof the at least one software module may be provided by an OS and theothers may be provided by a certain application.

FIG. 34 is a block diagram of the data learner 1310 according to someembodiments.

Referring to FIG. 34, the data learner 1310 may include a data obtainer1310-1, a pre-processor 1310-2, a learning data selector 1310-3, a modellearner 1310-4, and a model evaluator 1310-5.

The data obtainer 1310-1 may obtain data needed to detect the mainobject and the sub-object from the first image. The data obtainer 1310-1may obtain data needed to remove, from the first image, the dataassociated with at least some area of the first image where thesub-object is located. The data obtainer 1310-1 may obtain data neededto restore the data-removed area such that at least a portion of themain object hidden by the sub-object is included.

The data obtainer 1310-1 may obtain at least one image stored in theimage acquisition device 1000. For example, the data obtainer 1310-1 mayobtain an image including the face of the user and images including thefaces of people associated with the user (e.g., family members, a lover,and relatives of the user), the images stored in the image acquisitiondevice 1000. As another example, the data obtainer 1310-1 may obtainimages associated with a region where the user is located (e.g., majorbuildings, sculptures, and natural landscapes of the region), the imagesstored in the image acquisition device 1000. As another example, thedata obtainer 1310-1 may obtain at least one image disclosed on theInternet (e.g., major buildings, sculptures, and natural landscapes ofthe region). As another example, the data obtainer 1310-1 may obtain atleast one image associated with general objects (e.g., a tree, a trashcan, a streetlamp, traffic lights, a stall, a human, and an animal), theat least one image disclosed on the Internet.

The pre-processor 1310-2 may pre-process the obtained data such that theobtained data may be used to detect at least one of the main object andthe sub-object from the first image. The pre-processor 1310-2 maypre-process the obtained data such that the obtained data may be used toremove, from the first image, the data associated with at least somearea of the first image where the sub-object is located. Thepre-processor 1310-2 may pre-process the obtained data such that theobtained data may be used to restore the data-removed area such that atleast a portion of the main object hidden by the sub-object is included.

The pre-processor 1310-2 may process the obtained data into a presetformat such that the model learner 1310-4, which will be describedlater, may use the obtained data for learning for detecting at least oneof the main object and the sub-object from the first image.

The learning data selector 1310-3 may select data necessary for learningfrom among pieces of pre-processed data. The selected data may beprovided to the model learner 1310-4.

The learning data selector 1310-3 may select the data necessary forlearning from among the pieces of pre-processed data, according to apreset standard for detecting at least one of the main object and thesub-object from the first image. The learning data selector 1310-3 mayselect the data necessary for learning from among the pieces ofpre-processed data, according to a preset standard for removing, fromthe first image, the data associated with at least some area of thefirst image where the sub-object is located. The learning data selector1310-3 may select the data necessary for learning from among the piecesof pre-processed data, according to a preset standard for restoring thedata-removed area such that at least a portion of the main object hiddenby the sub-object is included.

The learning data selector 1310-3 may select data according to astandard previously set due to learning by the model learner 1310-4,which will be described later.

The model learner 1310-4 may learn a standard about how to detect atleast one of the main object and the sub-object from the first image,based on learning data. The model learner 1310-4 may learn a standardabout how to remove, from the first image, the data associated with atleast some area of the first image where the sub-object is located,based on learning data. The model learner 1310-4 may learn a standardabout how to restore the data-removed area such that at least a portionof the main object hidden by the sub-object is included, based onlearning data.

The model learner 1310-4 may learn a standard about which learning datais to be used in order to detect at least one of the main object and thesub-object from the first image. The model learner 1310-4 may learn astandard about which learning data is to be used in order to remove,from the first image, the data associated with at least some area of thefirst image where the sub-object is located. The model learner 1310-4may learn a standard about which learning data is to be used in order torestore the data-removed area such that at least a portion of the mainobject hidden by the sub-object is included.

The model learner 1310-4 may train a trained model that is used todetect at least one of the main object and the sub-object from the firstimage, by using the learning data. The model learner 1310-4 may train atrained model that is used to remove, from the first image, the dataassociated with at least some area of the first image where thesub-object is located, by using the learning data. The model learner1310-4 may train a trained model that is used to restore thedata-removed area such that at least a portion of the main object hiddenby the sub-object is included, by using the learning data.

In this case, the trained model may be a pre-established model. Forexample, the trained model may be a model previously established byreceiving basic learning data (for example, sample data).

The trained model may be established considering, for example, anapplication field of a recognition model, a purpose of learning, or thecomputer performance of a device. The trained model may include, forexample, a model based on a neural network. For example, a model, suchas a DNN, an RNN, or a BRDNN, may be used as the trained model, butembodiments are not limited thereto.

According to various embodiments, when a plurality of trained modelsthat are pre-established exist, the model learner 1310-4 may determine atrained model having a high relationship between input learning data andbasic learning data as the trained model to be trained In this case, thebasic learning data may be pre-classified according to types of data,and the trained model may be pre-established according to the types ofdata. For example, the basic learning data may be pre-classifiedaccording to various standards such as an area where the learning datais generated, a time for which the learning data is generated, a size ofthe learning data, a genre of the learning data, a generator of thelearning data, and a type of the object in the learning data

The model learner 1310-4 may train the trained model by using a learningalgorithm including, for example, error back-propagation or gradientdescent.

The model learner 1310-4 may train the trained model through supervisedlearning by using, for example, the learning data as an input value. Themodel learner 1310-4 may train the trained model through unsupervisedlearning to find a standard for detecting at least one of a main objectand a sub-object from the first image, by detecting at least one of amain object and a sub-object without supervision and self-learning atype of data needed to provide a response operation corresponding to aresult of the detection. The model learner 1310-4 may train the trainedmodel through reinforcement learning using a feedback about whether aresult of the detection of at least one of a main object and asub-object according to learning is right.

When the trained model is trained, the model learner 1310-4 may storethe trained model. In this case, the model learner 1310-4 may store thetrained model in a memory of a device including the data recognizer1320. Alternatively, the model learner 1310-4 may store the trainedmodel in a memory of a server that is connected with the device via awired or wireless network.

In this case, the memory in which the trained model is stored may alsostore, for example, a command or data related with at least one othercomponent of the device. The memory may also store software and/or aprogram. The program may include, for example, a kernel, a middleware,an application programming interface (API), and/or an applicationprogram (or an application).

When the model evaluator 1310-5 inputs evaluation data to the trainedmodel and a recognition result that is output from the evaluation datadoes not satisfy a predetermined standard, the model evaluator 1310-5may enable the model learner 1310-4 to learn again. In this case, theevaluation data may be preset data for evaluating the trained model.

For example, when the number or percentage of pieces of evaluation datathat provide inaccurate recognition results from among recognitionresults of the trained model with respect to the evaluation data exceedsa preset threshold, the model evaluator 1310-5 may evaluate that thepredetermined standard is not satisfied. For example, when thepredetermined standard is defined as 2% and the trained model outputswrong recognition results for more than 20 pieces of evaluation datafrom among a total of 1000 pieces of evaluation data, the modelevaluator 1310-5 may evaluate that the trained model is not appropriate.

When there are a plurality of trained models, the model evaluator 1310-5may evaluate whether each of the plurality of trained models satisfiesthe predetermined standard, and may determine, as a final trained model,a trained model that satisfies the predetermined standard. In this case,when a plurality of trained models satisfy the predetermined standard,the model evaluator 1310-5 may determine one or a predetermined numberof trained models that are preset in a descending order of evaluationscores as final trained models.

At least one of the data obtainer 1310-1, the pre-processor 1310-2, thelearning data selector 1310-3, the model learner 1310-4, and the modelevaluator 1310-5 in the data learner 1310 may be manufactured in theform of at least one hardware chip and may be mounted on a device. Forexample, at least one of the data obtainer 1310-1, the pre-processor1310-2, the learning data selector 1310-3, the model learner 1310-4, andthe model evaluator 1310-5 may be manufactured in the form of adedicated hardware chip for AI, or may be manufactured as a portion ofan existing general-purpose processor (for example, a CPU or an AP) or aprocessor dedicated to graphics (for example, a GPU) and may be mountedon any of the aforementioned various devices.

The data obtainer 1310-1, the pre-processor 1310-2, the learning dataselector 1310-3, the model learner 1310-4, and the model evaluator1310-5 may be all mounted on a single device, or may be respectivelymounted on independent devices. For example, some of the data obtainer1310-1, the pre-processor 1310-2, the learning data selector 1310-3, themodel learner 1310-4, and the model evaluator 1310-5 may be included ina device, and the others may be included in a server.

For example, at least one of the data obtainer 1310-1, the pre-processor1310-2, the learning data selector 1310-3, the model learner 1310-4, andthe model evaluator 1310-5 may be implemented as a software module. Whenat least one of the data obtainer 1310-1, the pre-processor 1310-2, thelearning data selector 1310-3, the model learner 1310-4, and the modelevaluator 1310-5 is implemented as a software module (or a programmodule including instructions), the software module may be stored in anon-transitory computer-readable recording medium. In this case, the atleast one software module may be provided by an OS or by a certainapplication. Alternatively, some of the at least one software module maybe provided by an OS and the others may be provided by a certainapplication.

The processor 1300 may use various trained models, and may efficientlylearn a standard for generating at least one of a main object and asub-object from the first image according to various methods via thevarious trained models.

FIG. 35 is a block diagram of the data recognizer 1320 according to someembodiments.

Referring to FIG. 35, the data recognizer 1320 may include a dataobtainer 1320-1, a pre-processor 1320-2, a recognition data selector1320-3, a recognition result provider 1320-4, and a model refiner1320-5.

The data obtainer 1320-1 may obtain data needed to detect the mainobject and the sub-object from the first image. The data obtainer 1320-1may obtain data needed to remove, from the first image, the dataassociated with at least some area of the first image where thesub-object is located. The data obtainer 1320-1 may obtain data neededto restore the data-removed area such that at least a portion of themain object hidden by the sub-object is included.

The pre-processor 1310-2 may pre-process the obtained data such that theobtained data may be used to detect at least one of the main object andthe sub-object from the first image. The pre-processor 1310-2 maypre-process the obtained data such that the obtained data may be used toremove, from the first image, the data associated with at least somearea of the first image where the sub-object is located. Thepre-processor 1310-2 may pre-process the obtained data such that theobtained data may be used to restore the data-removed area such that atleast a portion of the main object hidden by the sub-object is included.

The pre-processor 1320-2 may process the obtained data into a presetformat such that the recognition result provider 1320-4, which will bedescribed later, may use the data obtained to detect at least one of themain object and the sub-object from the first image. The pre-processor1320-2 may pre-process the obtained data into a preset format such thatthe recognition result provider 1320-4 may use the data obtained toremove, from the first image, the data associated with at least somearea of the first image where the sub-object is located.

The recognition data selector 1320-3 may select data needed to detect atleast one of the main object and the sub-object from the first image,from among pieces of pre-processed data. The recognition data selector1320-3 may select data needed to remove, from the first image, the dataassociated with at least some area of the first image where thesub-object is located, from among the pieces of pre-processed data. Therecognition data selector 1320-3 may select data needed to restore thedata-removed area such that at least a portion of the main object hiddenby the sub-object is included, from among the pieces of pre-processeddata. The selected data may be provided to the recognition resultprovider 1320-4.

The recognition data selector 1320-3 may select some or all of thepieces of pre-processed data, according to a preset standard fordetecting at least one of the main object and the sub-object from thefirst image. The recognition data selector 1320-3 may select some or allof the pieces of pre-processed data, according to a preset standard forremoving, from the first image, the data associated with at least somearea of the first image where the sub-object is located. The recognitiondata selector 1320-3 may select some or all of the pieces ofpre-processed data, according to a preset standard for restoring thedata-removed area such that at least a portion of the main object hiddenby the sub-object is included.

The recognition data selector 1320-3 may select data according to astandard previously set due to learning by the model learner 1310-4,which will be described later.

The recognition result provider 1320-4 may detect at least one of themain object and the sub-object from the first image, by applying theselected data to a data recognition model. The recognition resultprovider 1320-4 may remove, from the first image, the data associatedwith at least some area of the first image where the sub-object islocated, by applying the selected data to the data recognition model.The recognition result provider 1320-4 may restore the data-removed areasuch that at least a portion of the main object hidden by the sub-objectis included. by applying the selected data to the data recognitionmodel.

The recognition result provider 1320-4 may provide a recognition resultthat conforms to a data recognition purpose. The recognition resultprovider 1320-4 may apply the selected data to the data recognitionmodel by using the data selected by the recognition data selector 1320-3as an input value. The recognition result may be determined by the datarecognition model. For example, a recognition result of detecting atleast one of the main object and the sub-object from the first image, arecognition result of removing, from the first image, the dataassociated with at least some area of the first image where thesub-object is located, and a recognition result of restoring thedata-removed area such that at least a portion of the main object hiddenby the sub-object is included may be provided as text, an image, orinstructions (e.g., application execution instructions or modulefunction execution instructions).

The model refiner 1320-5 may enable the data recognition model to beupdated, based on an evaluation of a recognition result provided by therecognition result provider 1320-4. For example, the model refiner1320-5 may enable the model learner 1310-4 to update the datarecognition model, by providing the recognition result provided by therecognition result provider 1320-4 to the model learner 1310-4.

At least one of the data obtainer 1320-1, the pre-processor 1320-2, therecognition data selector 1320-3, the recognition result provider1320-4, and the model refiner 1320-5 within the data recognizer 1320 maybe manufactured in the form of at least one hardware chip and may bemounted on a device. For example, at least one of the data obtainer1320-1, the pre-processor 1320-2, the recognition data selector 1320-3,the recognition result provider 1320-4, and the model refiner 1320-5 maybe manufactured in the form of a dedicated hardware chip for AI, or maybe manufactured as a portion of an existing general-purpose processor(for example, a CPU or an AP) or a processor dedicated to graphics (forexample, a GPU) and may be mounted on any of the aforementioned variousdevices.

The data obtainer 1320-1, the pre-processor 1320-2, the recognition dataselector 1320-3, the recognition result provider 1320-4, and the modelrefiner 1320-5 may be all mounted on a single electronic apparatus, ormay be respectively mounted on independent electronic apparatuses. Forexample, some of the data obtainer 1320-1, the pre-processor 1320-2, therecognition data selector 1320-3, the recognition result provider1320-4, and the model refiner 1320-5 may be included in an electronicapparatus, and the others may be included in a server.

At least one of the data obtainer 1320-1, the pre-processor 1320-2, therecognition data selector 1320-3, the recognition result provider1320-4, and the model refiner 1320-5 may be implemented as a softwaremodule. When at least one of the data obtainer 1320-1, the pre-processor1320-2, the recognition data selector 1320-3, the recognition resultprovider 1320-4, and the model refiner 1320-5 is implemented as asoftware module (or a program module including instructions), thesoftware module may be stored in a non-transitory computer-readablerecording medium. In this case, the at least one software module may beprovided by an OS or by a certain application. Alternatively, some ofthe at least one software module may be provided by an OS and the othersmay be provided by a certain application.

The image acquisition device 1000 may provide the user with an imagethat conforms to an intention of the user by using a trained model towhich a learned result has been applied.

FIG. 36 is a block diagram illustrating an example where the imageacquisition device 1000 and the server 2000 interoperate to learn andrecognize data, according to some embodiments.

Referring to FIG. 36, the server 2000 may learn a standard for detectingat least one of the main object and the sub-object from the first image,a standard for removing, from the first image, the data associated withat least some area of the first image where the sub-object is located,and a standard for restoring the data-removed area such that at least aportion of the main object hidden by the sub-object is included, and theimage acquisition device 1000 may detect at least one of the main objectand the sub-object from the first image, remove, from the first image,the data associated with at least some area of the first image where thesub-object is located, and restore the data-removed area such that atleast a portion of the main object hidden by the sub-object is included,based on a learning result of the server 2000.

In this case, a model learner 2340 of the server 2000 may perform afunction of the data learner 1310 of FIG. 33.

The model learner 2340 of the server 2000 may learn the standard fordetecting the main object and the sub-object from the first image. Themodel learner 2340 may learn a standard about which data is to be usedin order to detect the main object and the sub-object from the firstimage. The model learner 2340 may learn the standard for detecting themain object and the sub-object from the first image, by obtaining datafor use in learning and applying the obtained data to a data recognitionmodel which will be described later.

The model learner 2340 may learn the standard for removing, from thefirst image, the data associated with at least some area of the firstimage where the sub-object is located. The model learner 2340 may learna standard about which data is to be used in order to remove, from thefirst image, the data associated with at least some area of the firstimage where the sub-object is located. The model learner 2340 may learnthe standard for removing, from the first image, the data associatedwith at least some area of the first image where the sub-object islocated, by obtaining data for use in learning and applying the obtaineddata to the data recognition model which will be described later.

The model learner 2340 may learn the standard for restoring thedata-removed area such that at least a portion of the main object hiddenby the sub-object is included. The model learner 2340 may learn astandard about which data is to be used in order to restore thedata-removed area such that at least a portion of the main object hiddenby the sub-object is included. The model learner 2340 may learn thestandard for restoring the data-removed area such that at least aportion of the main object hidden by the sub-object is included, byobtaining data for use in learning and applying the obtained data to thedata recognition model which will be described later.

The model learner 2340 may learn at least one of the standard fordetecting at least one of the main object and the sub-object from thefirst image, the standard for removing, from the first image, the dataassociated with at least some area of the first image where thesub-object is located, and the standard for restoring the data-removedarea such that at least a portion of the main object hidden by thesub-object is included, by obtaining data for use in learning andapplying the obtained data to a trained model which will be describedlater.

The recognition result provider 1320-4 of the image acquisition device1000 may detect at least one of the main object and the sub-object fromthe first image, remove, from the first image, data associated with atleast some area of the first image where the sub-object is located, andrestore the data-removed area such that at least a portion of the mainobject hidden by the sub-object is included, by applying the dataselected by the recognition data selector 1320-3 to the data recognitionmodel generated by the server 2000. For example, the recognition resultprovider 1320-4 may transmit the data selected by the recognition dataselector 1320-3 to the server 2000, and the server 2000 may request todetect at least one of the main object and the sub-object from the firstimage, remove, from the first image, the data associated with at leastsome area of the first image where the sub-object is located, andrestore the data-removed area such that at least a portion of the mainobject hidden by the sub-object is included, by applying the dataselected by the recognition data selector 1320-3 to a recognition model.

The recognition result provider 1320-4 may receive, from the server2000, information about a method of detecting at least one of the mainobject and the sub-object from the first image, removing, from the firstimage, the data associated with at least some area of the first imagewhere the sub-object is located, and restoring the data-removed areasuch that at least a portion of the main object hidden by the sub-objectis included.

Alternatively, the recognition result provider 1320-4 of the imageacquisition device 1000 may receive the recognition model generated bythe server 2000 from the server 2000, and may detect at least one of themain object and the sub-object from the first image by using thereceived recognition model. The recognition result provider 1320-4 mayremove, from the first image, the data associated with at least somearea of the first image where the sub-object is located, and restore thedata-removed area such that at least a portion of the main object hiddenby the sub-object is included, by using the received recognition model.In this case, the recognition result provider 1320-4 of the imageacquisition device 1000 may detect at least one of the main object andthe sub-object from the first image, remove, from the first image, thedata associated with at least some area of the first image where thesub-object is located, and restore the data-removed area such that atleast a portion of the main object hidden by the sub-object is included,by applying the data selected by the recognition data selector 1320-3 tothe recognition model received from the server 2000.

The various units of the image acquisition device 1000 may beimplemented by a combination of a processor, memory, and program codelocated in the memory and executed by the processor to perform thevarious functions of the methods and devices described above.

The various units of the server 2000 may be implemented by a combinationof a processor, memory, and program code located in the memory andexecuted by the processor to perform the various functions of themethods and devices described above.

The image acquisition device 1000 and the server 2000 may effectivelydistribute and perform operations for learning of a trained model anddata recognition, and accordingly efficiently perform data processing inorder to provide a service that is consistent with a user's intention,and effectively protect user's privacy.

Some embodiments may also be embodied as a storage medium includinginstruction codes executable by a computer such as a program moduleexecuted by the computer. A computer readable medium can be anyavailable medium which can be accessed by the computer and includes allvolatile/non-volatile and removable/non-removable media. Further, thecomputer readable medium may include all computer storage andcommunication media. The computer storage medium includes allvolatile/non-volatile and removable/non-removable media embodied by acertain method or technology for storing information such as computerreadable instruction code, a data structure, a program module or otherdata.

The terminology “˜unit” used herein may be a hardware component such asa processor or a circuit, and/or a software component that is executedby a hardware component such as a processor.

Although the embodiments of the present disclosure have been disclosedfor illustrative purposes, one of ordinary skill in the art willappreciate that diverse variations and modifications are possible,without departing from the spirit and scope of the disclosure. Thus, theabove embodiments should be understood not to be restrictive but to beillustrative, in all aspects. For example, respective elements describedin an integrated form may be dividedly used, and the divided elementsmay be used in a state of being combined.

While one or more example embodiments have been described with referenceto the figures, it will be understood by those of ordinary skill in theart that various changes in form and details may be made therein withoutdeparting from the spirit and scope as defined by the following claims.

What is claimed is:
 1. A method of providing an image using a trainedartificial intelligence (AI) model, the method comprising: obtaining,using a camera of an electronic device, a plurality of images includinga first image that includes a plurality of objects; identifying a firstobject, which represents a first human, as a main object among theplurality of objects included in the first image; detecting a movementof a second object, which represents a second human, based on aplurality of positions of the second object in at least two images ofthe plurality of images, the second object being different from thefirst object; based on the detecting of the movement of the secondobject, determining, by a first trained AI model stored in a memory ofthe electronic device, the second object as an object for removal fromthe first image; obtaining a second image including the first object bygenerating data for an area of the first image corresponding to thesecond object in order to remove the second object from the first image,wherein the generated data corresponds to a background hidden by thesecond object; and displaying the obtained second image using a displayof the electronic device.
 2. The method of claim 1, wherein the secondobject is a moving object.
 3. The method of claim 1, wherein theobtaining the second image comprises generating the second image byremoving a first image data of the area of the first image correspondingto the second object.
 4. The method of claim 1, wherein the obtaining ofthe second image comprises generating the second image by replacing afirst image data of the area of the first image corresponding to thesecond object with second image data.
 5. The method of claim 1, whereinthe obtaining of the second image comprises generating, by a secondtrained AI model, the second image including the first object byremoving the second object from the first image.
 6. An image acquisitiondevice comprising: a display; a camera; a memory configured to store oneor more instructions, at least one processor configured to execute theone or more instructions to: control the camera to obtain a plurality ofimages including a first image including a plurality of objects;identify a first object, which represents a first human, as a mainobject among the plurality of objects included in the first image;detect a movement of a second object, which represents a second human,based on a plurality of positions of the second object in at least twoimages of the plurality of images, the second object being differentfrom the first object; based on the detecting of the movement of thesecond object, determine, by a first trained artificial intelligence(AI) model stored in the memory, the second object as an object forremoval from the first image; obtain a second image including the firstobject by generating data for an area of the first image correspondingto the second object in order to remove the second object from the firstimage, wherein the generated data corresponds to a background hidden bythe second object; and control the display to display the obtainedsecond image.
 7. The image acquisition device of claim 6, wherein thesecond object is a moving object.
 8. The image acquisition device ofclaim 6, wherein the at least one processor is further configured toexecute the one or more instructions to: generate the second image byremoving a first image data of the area of the first image correspondingto the second object.
 9. The image acquisition device of claim 6,wherein the at least one processor is further configured to execute theone or more instructions to: generate the second image by replacing afirst image data of the area of the first image corresponding to thesecond object with second image data.
 10. The image acquisition deviceof claim 6, wherein the at least one processor is further configured toexecute the one or more instructions to: generate, by a second trainedAI model, the second image including the first object by removing thesecond object from the first image.